Skip to content

Bug Fix and use of new GPUCompiler persistent caching (8x improvement to loading time)#760

Closed
collinwarner wants to merge 2 commits intoEnzymeAD:mainfrom
collinwarner:main
Closed

Bug Fix and use of new GPUCompiler persistent caching (8x improvement to loading time)#760
collinwarner wants to merge 2 commits intoEnzymeAD:mainfrom
collinwarner:main

Conversation

@collinwarner
Copy link
Copy Markdown

@collinwarner collinwarner commented Apr 23, 2023 •
edited
Loading

Add functionality of GPUCompiler JuliaGPU/GPUCompiler.jl#425. Sees 8x improvement of cached loading times of the enzyme package.

Additionally latent bug found and fixed related to precompilation of Enzyme functions

@collinwarner
Copy link
Copy Markdown
Author

collinwarner commented Apr 23, 2023 •
edited
Loading

image

@collinwarner collinwarner mentioned this pull request Apr 23, 2023
Draft
@collinwarner collinwarner changed the title add GPUCompiler persistent caching Bug Fix and use of new GPUCompiler persistent caching May 3, 2023
@collinwarner
Copy link
Copy Markdown
Author

collinwarner commented May 3, 2023 •
edited
Loading

Bug Found:

The following script will error out with a segfault in the current Enzyme implementation. This occurs because the generated functions for thunk and deferred_codegen do no retrigger GPUCompiler and instead use stale values. Simply dropping the generated tag fixes the segfault but we are no longer able to look up unique method instances as the returned world ages are incorrect. Instead we should switch to using GPUCompiler.methodinstance directly dropping the need to filter on world ages.

module EnzymeTest                                                               
using Enzyme                                                                    
                                                                                
f1(x) = x*x                                                                     
                                                                                
@show autodiff(Reverse, f1, Active(1.0))                                        
                                                                                
                                                                                
function __init__()                                                             
    @show autodiff(Reverse, f1, Active(1.0))                                    
end                                                                             
                                                                                
end # module EnzymeTest

With the following segfault:


julia> using EnzymeTest
[ Info: Precompiling EnzymeTest [87ed9522-ec2b-4441-9f14-f86404eed9ba]
autodiff(Reverse, f1, Active(1.0)) = ((2.0,),)

[1013783] signal (11.1): Segmentation fault
in expression starting at REPL[1]:1
unknown function (ip: 0x7f86ad477020)
macro expansion at /home/collinw/.julia/dev/Enzyme/src/compiler.jl:8620 [inlined]
enzyme_call at /home/collinw/.julia/dev/Enzyme/src/compiler.jl:8312 [inlined]
CombinedAdjointThunk at /home/collinw/.julia/dev/Enzyme/src/compiler.jl:8275 [inlined]
autodiff at /home/collinw/.julia/dev/Enzyme/src/Enzyme.jl:205 [inlined]
autodiff at /home/collinw/.julia/dev/Enzyme/src/Enzyme.jl:228 [inlined]
autodiff at /home/collinw/.julia/dev/Enzyme/src/Enzyme.jl:214 [inlined]
macro expansion at ./show.jl:1153 [inlined]
__init__ at /home/collinw/julia_master/EnzymeTest/src/EnzymeTest.jl:10
Allocations: 2883 (Pool: 2874; Big: 9); GC: 0
Segmentation fault (core dumped)

@collinwarner collinwarner changed the title Bug Fix and use of new GPUCompiler persistent caching Bug Fix and use of new GPUCompiler persistent caching (8x improvement to loading time) May 4, 2023
@vchuravy
Copy link
Copy Markdown
Member

vchuravy commented May 4, 2023

Can you split that last change out into a new PR?

@collinwarner
Copy link
Copy Markdown
Author

collinwarner commented May 4, 2023

Changing to new PR, cleaning

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@collinwarner @vchuravy